PR #64 Response to Architectural Feedback

@vjmp - I owe you a proper response to the questions you raised on #66. Let me address each one directly, with benchmark data from a controlled test environment (Python 3.12, rpaframework, robocorp-browser, requests - 12,038 files, 237MB compressed).

Benchmark Results

Where LIFT time actually goes:

micromamba:        27.1s  (60%)
uv install:         9.4s  (21%)
Record to hololib:  1.7s  (4%)
Restore to space:   5.5s  (12%)
Other:              1.7s  (3%)

The honest reality: 86% of LIFT time is in external package managers. RCC controls about 8 seconds of a 45 second build.

Addressing Your Questions from #66

1. Why is additional speed important? And on which use-case?

Target use-case: First-time LIFT on a developer machine.

Not targeting:

• DROP (already 0.66s - sub-second is fast enough)
• Enterprise farms with pre-built catalogs (they skip LIFT entirely)
• Additional environments with shared dependencies (already fast due to hololib deduplication)

Who benefits:

• Developers setting up new machines
• CI/CD pipelines doing fresh builds
• Users trying rcc for the first time

The 4 second improvement on LIFT is a 47% speedup on the RCC-controlled portion, but only 8% on overall user experience because micromamba and uv dominate.

2. Which is more important, speed or simplicity of algorithm?

Simplicity. You are right to push back here.

The zstd format change itself is simple:

• Magic byte detection (4 bytes: 0x28 0xB5 0x2F 0xFD for zstd, 0x1F 0x8B for gzip)
• Dual codec support with automatic detection
• Backward compatible - old gzip files still work

What I added that is NOT simple: the prefetching system (LRU cache, adaptive depth, hit rate tracking, backpressure). I am reconsidering whether to keep it.

3. Does this new format support parallel/concurrent access?

Yes, unchanged from current design:

• Still uses atomic .part + rename pattern
• Still uses existing pathlib.Locker for space coordination
• No shared state between processes
• Each file in hololib is still independent

The zstd change does not alter the concurrency model at all.

4. Can you keep this algorithm in your head?

The zstd format detection and dual-read: yes.

The prefetching system (4 workers, LRU, adaptive depth, backpressure): no, honestly.

This is a strong signal that I should remove the prefetching code.

5. When new environments add just few files, how accessing them will be done?

Unchanged. This PR does not introduce packfiles.

Filesystem is still the index. File existence check is still: does hololib/library/XX/YYYYY... exist? Direct lookup, direct data access. The zstd change only affects how individual files are compressed, not how they are organized or discovered.

6. Does this reduce or increase disk space usage?

Neutral. Benchmark shows 237MB with both gzip and zstd for the same environment. Zstd at default compression level produces similar sizes to gzip for this workload (mostly Python bytecode, text files, small binaries).

7. Does this remove new plan for GC?

No. GC is completely unaffected. Files are still individual, still addressed by hash, still cleaned up the same way.

8. If something corrupts one byte in middle of pack, does it invalidate all environments?

Not applicable to this PR. This PR does not introduce packfiles.

Each file in hololib remains independent. If one file is corrupted, only environments that need exactly that file are affected. The corruption blast radius is unchanged from current behavior.

I explored packfiles in issue #66 as research, but concluded the complexity does not justify the gains - especially given your concerns about long-term maintainability. I am not pursuing packfiles.

9. If there is custom pack file format, how to tell enterprises how to check content?

Not applicable. No packfiles in this PR.

Files are still "just zstd-compressed files" (or gzip for older catalogs). Standard tools can inspect them:

• zstd -d < file to decompress
• file command detects the format
• Magic byte detection is well-documented

Enterprises can still audit the hololib the same way they do today.

10. What happens when pack files are accessed concurrently?

Not applicable. No packfiles.

Concurrent access model is unchanged: multiple processes can read from hololib simultaneously, writes are coordinated via atomic .part + rename, and pathlib.Locker handles space-level coordination.

11. Is it ok that same hash/identity is on multiple pack files?

Not applicable. No packfiles in this PR.

The identity model is unchanged: one hash = one file in hololib/library/XX/YYYYYY.... No duplication, no ambiguity about which copy to use.

On Your Design Philosophy Notes

You wrote:

some optimizations were not done, because I valued maintenance simplicity over absolute speed (thinking on long game)

I understand and respect this. The long game matters more than benchmarks.

filesystem is both index and data storage directly (direct lookup, direct data access)

This PR preserves this completely. No separate index files, no packfile lookup tables. The filesystem path IS the lookup key.

if one file is corrupted, only those environments that have exactly that file cannot be restored anymore

Still true. No change to corruption isolation.

more environments get "stacked" on one hololib, less space each new environment will take on disk

Still true. Deduplication works the same way.

What This PR Keeps

1. zstd compression - Simple change, backward compatible, 4 second improvement on LIFT
2. Decoder/encoder pools - Low complexity via sync.Pool, measurable speedup
3. Race condition fixes - The symlink and .part file handling were real bugs I hit during testing

What I Should Remove

The prefetching system. Based on your feedback and my own inability to "keep it in my head," the complexity is not justified. Profiling shows syscalls are 39% of CPU time - prefetching reorders reads but does not reduce syscall count.

Proposed Path Forward

Ship this PR with:

• zstd compression (simple, backward compatible)
• Codec pools (low complexity)
• Bug fixes (real issues)

Remove:

• Prefetching system (too complex for the gain)

Close issue #66 as "research complete, not pursuing packfiles."

What do you think? I want to make sure any changes are net-positive for long-term maintainability, and I hear your concerns about complexity.